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Analyzing Content Coverage and Emphasis: 
A Study of Three Curricula and Two Tests 

Theoretical Framework 

The conclusion emerging from recent studies of the relationship 
between classroom processes and student achievement is that "major 
factors in the process of knowledge acquisition in the classroom are 
the content and emphasis of the curriculum in use" (Berliner & 
Rosenshine, 1976). For exac^le, studies by Annento (1975), Chang and 
Raths (1971), Rosenshine (1968), and Shutes (1969) found significant 
positive correlations betv.-een content covered and achievement. The 
related variable of content emphasis was found to be correlated vrith 
achievement in studies reviewed by IJalker and Schaffarzick (1574) and 
in a major study of 166 low SES classroou^s (Stallings i Kaskowitz, 1974). 
Such studies show that content coverage and emphasis are at least as 
important as teaching method for student achievement. Yet research 
analyzing and conTparing the content of various curricula in the 
attempt to discover specific variables affecting achievement is rare. 

Achievement Is a function of content coverage and emphasis of 
curricula, but the measures of achievement are in turn dependent on 
the particular content and emphasis of the instrument employed. As 
Walker and Schaffarzick (1974) shoi/, the demonstrated effectiveness 
of curricula (neasured by "achievement") is a function of the content 
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of the posttest: each curriculum in the studies reviewed seemed to 
be most effective when the posttest reflected the content emphasis of 
that curriculum. Therefore, achievement is most accurately defined 
with respect to curricula. To make judgments of effectiveness, re- 
searchers must know both what the curriculum teaches and whether the 
tests adequately measure what is taught. 

Objeccives 

The purpose of the present study was to analyze part of the content 
of three nationally used reading curricula and tvw common standardized 
tests in order to discover congruencies and incongruencies of content 
coverage and emphases among curricula, between tests, and between 
curricula and tests. The particular content of interest was reading 
comprehension. 

Methods 

Three reading curricula designed for the second half of the third 
grade were chosen for analysis: 

1) The Economy Company: The Mysterious Wisteria (Level 10), 1975. 

2) Ginn and Company: Reading 360: All Sorts of Things (Level 10), 
1969. 

3) Houghton Mifflin Company: Fiesta (Level 9), 1971. 

Two standardized tests judged to be in common use as measures of 
achievement in educational research were also selected (forms appro- 
priate for the third grade v/ere employed): 
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1) California Achievement Tests: Reading (Level 2, Form A), 
:ionterey, Calif.: CTB/Mc6raw-Hill , Inc., 1970. 

2) Metropolitan Achievement Tests: Form G . Hew York: Harcourt 
Brace Javanovicb, Inc., 1970. 

In order to have a measure of content coverage and emphasis com- 
parable across curricula and tests, it was decided to obtain frequen- 
cies of exercises from the curriculum materials and items from the 
tests in several subcategories of reading comprehension. Sixteen 
categories were adapted from SRA's SOBAR Reading Catalog of Objectives 
3-9 (1975) and were operationally defined as follows: 

Detail : The reader answers questions that either have the exact 
same surface form as single text sentences or that have the same surface 
form except for pronoun substitutions , 

Paraphrase Level 1 : The reaaer ans^wers questions that have 
different surface forms but the same meaning as single text sentences; 
i.e., questions and text sentences match except for synonym substitutions. 

Paraphrase Level 2 : The reader ansv/ers questions that have 
different surface forms but tL2 same meaning as two or more text 
sentences. 

Cloze Sentences : Given a sentence with a word deleted, the reader 
selects the appropriate word from several alternatives. 

Classifying : Given a set of reading passages, the reader identi- 
fies similar passages according to some criterion. 
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Following Directions : Given a set of written directions, the 
reader p'er forms the indicated task. 

Sequence , The reader orders presented events into a sequence 
Hatching the presented sequence. 

Drawing Conclusions : The reader answers questions, calling for a 
conclusion based on the material in the reading selection. 

Ha in Idea: The reader answers questions calling for identifica- 
tion of the main idea and/or an appropriate title. 

Supporting Information : The reader identifies or supplies 
subordinate topics given the main topic in a content outline. 

Cause and Effect : The reader answers questions concerning 
cause-and-ef feet relationships. 

Words in Context : Given a sentence containing context clues to 
the meaning of an unknown word, the reader selects the appropriate 
Meaning from two or more alternatives. 

Figurative Language : The reader identifies similes and metaphors 
in a reading selection. 

Fantasy - Reality : The reader identifies a reading selection 
as either a representation of fantasy or reality. 

ilood - Setting : The reader answers questions concerning the 
mood or setting of a reading selection. 

Character's Emotions and Traits : The reader answers questions 
concerning the emotions and traits of characters in a reading selection. 
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The authors classified and recorded frequencies of reading 
comprehension items in the curriculuia materials (teacher's manual and 
workbook) and tests. For the curricula, only written exercises designed 
to be cornpleted by all students were coded. Oral exercises, questions 
fiieant for class discussion, and individualized activities were not 
coded because these exercises typically do not involve every student. 
An interrater reliability (percent agreement) of .61 was established 
on the basis of three blocks of forty items randomly selected from the 
three curricula. Since this reliability is acceptably high, the final 
iton counts were taken to be the average of the item counts of the 
three raters. 

Results and Conclusions 

The texts differ widely in their relative emphasis of reading 
comprehension in general (Table 1) and of particular reading compre- 
hension categories (Table 2). Economy stresses cloze sentences » 
supporting information, and sequence; Ginn gives by far the most 
emphasis to clczs sentences; and Houghton-Mifflin emphasizes words in 
context and clozs ser.tences. Table 3 presents another v/ay of looking 
at the differences among texts. Detail, paraphrase level 1, paraphrase 
level cloze sentences, and following directions were judged to be 
literal coi.v^ehansion items, while all ot'ier categories appeared to 
involve iaferential comprehension. According to this classification, 
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Economy and Houghton-I4ifflin give almost the same relative emphasis to 
literal and inferential comprehension, but Ginn emphasizes literal 
items mach more, largely because of its heavy emphasis (about half of 
the items) on cloze sentences. 

Insert Tables 1, 2, and 3 about here 

Correlations were computed (based on rank order of emphasis given 
to different categories) between the curricula based on all the cate- 
gories (Table 4) and also on the six categories the curricula had in 
ccKimon with the two tests (detail, paraphrase level 1, paraphrase level 
2, drawing conclusions, main idea, and character's emotions and traits) 
(Table 5). Economy and Houghton-Hifflin were moderately correlated In 
both tables (r^ = .4), but the correlations between Economy and Ginn 
and Ginn and Houghton-Mifflin vary from low positive to low negative 
depending on whether one uses all the reading comprehension categories 
(Table 4) or just the main categories used in the reading comprehension 
tests (Table 5). Cased on these data, the three curricula appear 
moderately to highly distinct. 

Insert Tables 4 and 5 about here 

The two standardized tests are quite similar in relative emphasis 
of reading comprehension, yielding a high positive rank^order correlation 
(+.93). For the six categories cofranon to all texts and tests, the rank 
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order correlations between texts and tests were low (Table 5). The 
Ginn series and the IIAT correlated only +.10; the highest correlations 
were between Hoaghton-Hifflin and the two achievement tests, but even 
these correlations were in the low .40's. This suggests that what is 
emphasized on the tests is different from what is emphasized in the 
texts . 

Insert Table 6 about here 

A further finding reflecting differences between texts and tests 
is the fact that a large percentage cf the comprehension items taught 
are not tested on the standardized tests (Table 2). Out of the 16 
reading cowiprehension categories that are covered in one or more texts, 
only six are tested on the MAT snd seven on the CAT. From another 
perspective, 64% of the Economy, 65% of the Ginn, and 79% of the 
Houghton-ilifflin reading comprehension items do not have counterparts 
Oil the standardized tests. 

Approximately tv;o-thirds of the reading comprehension items on 
both the ilAT and CAT vjere categorized as detail, paraphrase level 1, 
and paraphrase level 2. According to our classification, these 
categories involve literal cor^prehension. As seen from Table 3, 
however, two of the three texts (Economy and Houghton-Mifflin) stress 
inferential categories • Detail, paraphrase level 1, and paraphrase 
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level 2 comprise only 18% of Economy's, 28% of Ginn's, and 13% of 
Houghton-Hifflin 's items. 

Discussion 

Jhe most significant finding of this study is the large discrepancy 
between reading comprehension skills taught and tested. Only a small 
percentage of skills emphasized in the curricula have counterparts on 
the standardized tests. Furthermore, the skills In the tests tend to 
be factual items entailing locating information in the presented text, 
whereas two out of three curricula give heavier emphasis to comprehen- 
sion skills that appear to require inference, interpretation, identi- 
fication of relationships, and synthesis. 

The study also revealed high variation among the texts but high 
consistency between the tests for relative content coverage. 

Judging from the correlations between texts and tests, Houghton- 
ilifflin v/as more related to the standardized tests than Ginn or Economy, 
Such a table of correlations might enable one to determine which 
curriculum is the most appropriate preparation for a particular stan- 
dardized test, or alternatively, which test is the best measure of the 
material covered in a particular curriculum. However, such use of a 
table of correlations is not yet justifiable, for we do not presently 
know v/hether the comprehension categories are psychologically distinct. 
If they are, then the categories v^hich are taught and probably the 
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sequence of training becomes important. However, it may be that 
reading comprehension is a global or general ability, with no component 
subskills. In that case, the choice of particular exercises would be 
irrelevant; all are equally suited to developing and testing a general 
reading comprehension ability. In sum, we do not yet know whether a 
mismatch between what is taught and v/hat is tested in reading comprehen- 
sion has any practical significance. 

Despite unanswered questions, the present study is important in 
its demonstration of a feasible methodology for addressing a long- 
neglected research problem— determining content coverage and content 
emphasis of both curricula and tests. Flore such studies comparing 
curricula and tests in different content areas and grade levels are 
needed. 
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Table 1 

/!e1atl¥C E.-nph4Sis on Reading and Reading Cocvprehension 
for Three Curricula and Two Tests 

Curricula Tests 
H<xighton- 

Econor^y Ginn riifflin .".AT CAT 

Total reading cicrcises 3.060 2,425 1,134 145 85 
Total read In-; coptprchersion 

enerciscs 333 892 614 24 35 
Percent rc^^in^] cc»»yreh«n$ion 

exercises 27.2 36. C 5<M 16.6 41.2 
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Table 3 

Percentages of Literal and Inferential 
Itans for Three Curricula 

I tew type Econouv Ginn H- ighton-Mifflin 

Literal 421 33% 4U 

Inferential 582 \7% SOZ 
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Table 4 

Correlations (r^) Between Curricula for 
All Reading Comprehension Categories 



Economy Ginn Houghton-Mifflin 

Econony ■♦^1,00 ■♦^.31 ■♦^.40 

Ginn +.31 +1.00 +.33 

Houghton-iJifflin ♦.40 +.33 -H-OO 
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Table 5 

Correlations (r^) Between Curricula for Six Reading 
Comprehension Categories in Common with 
MAT and CAT 



Economy Ginn Houghton-Mifflin 

Economy +1.00 -.OS +.43 

Ginn -.08 +1.00 -.14 

Houghton-IIifflin +.43 -.14 +1.00 
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Table 6 

elations (r^) Between Curricula and 
for Six Cotimon Reading Comprehension 
Categories 



Tests 

Curricula JIAT CAT 

Economy .21 .37 

Ginn .10 .31 

Houghton-Hifflin .41 .43 
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